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ABSTRACT 

Problems that arise in applying cost effectiveness 
evaluations to instructional technologies are outlined in this paper, 
to explain that schools can draw on no available knowledge to 
determine whether the new instructional strategies will be more 
effective than traditional instructional approaches, once costs have 
been taken into account. The paper shows that most of the 
requirements for sound cost effectiveness analysis cannot 
satisfied, given our present knowledge of the educational 
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COST EFFECTIVENESS EVALUATION OF INSTRUCTIONAL TECHNOLOGY: TtlE PROBLEMS 

by Henry M. Levin* 

Introduction 

Over the last two decades our society has experienced a revolution in 
decision-making processes. Where government and business once chose strat- 
egies on the basis of very meager information and intuition, modern decision- 
making relies upon the use of abundant sources of data, sophisticated ana- 
lytical techniques, and high speed digital computers. Since most enter- 
prises operate with a limited budget, they wish to make decisions that 
maximize output for any given cost or, conversely, minimize the costs for any 
given outcome. Such goals have stimulated the' development of a set of tools 
for determining the probable costs and benefits of alternative management 
strategies. These methods are classified broadly under the term of cost- 
effectiveness techniques since they are designed to aid in choosing those 

A 

approaches which yield the best outcome for any given cost.__/ 



* Henry M. Levin is associate professor of education, Stanford University. 



/ An overview can be found in Thomas A. Goldman (ed.) Cost Effectiveness 
Analysis : New Approache s in Decision-Making (New York: Frederick A. 
Praeger, Inc., 1967). 
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Concurrent with the recent revolution in management science and its 
particular branch of cost-effectiveness analysis has been the proliferation 
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of new instructional technologies for the schools. Some of the best knov7n 

. 

of these ' techniques are: computer-assisted instruction, individually pro- 
grammed instruction, the responsive environment or "talking typewriter", 

I 

educational games, and that relatively old example of the new technology, 
educational television. 

At the same time our society has become increasingly frustrated by 
•the failure of the schools to teach or motivate large numbers of disad- 
vantaged youngsters. Complicating this failure has been the fact that 
spending additional money for so-called compensatory education has not 
improved to any significant degree these educational outcomes. Indeed, it 
appears that where there has existed a basic failure among traditional ed- 
ucational approaches, the simple expedients of spending more money on 
reduced class size and additional remedial services are inadequate for 
resurrection. So, increasingly in the post-Sputnik era the bankruptcy of 
the schools has led to many demands and proposals for substantial changes 
in the instructional process. 

Given these three coinciding developments, one might expect a sig- 
nificant interaction among them. That is, we might expect to find the new 
instructional technologies being evaluated for their relative efficiencies 
in producing educational outcomes; and the ones showing the greatest effec- 
tiveness relative to their costs would replace the allegedly less efficient 
traditional schooling processes. Paradoxically, this interaction has not 
taken place. The typical school continues to carry out its tasks in the 
same manner as it has always done. In fact, no reliable information on the 
relative costs and benefits of the new instructional technologies (or even 
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the old ones) has become available. Schools and school districts have no 

, .^ 4 . 

objective data with which they can make determinations based upon the 
relationships between costs and performances of alternative instructional 
strategies. 

Given this dearth of data, the schools have been very slow to adopt 
new techniques, and this conservatism has strong justification. There 
simply is no available knowledge that schools can draw upon to determine 
whether the new instructional approaches will be more effective, once costs 
are taken into account, than are traditional instructional approaches. 

The purpose of this paper is that of explaining this paradox by out” 
lining the problems that arise in applying cost-effectiveness evaluations 
to instructional technologies. It will be shown that most of the require- 
ments for sound cost-effectiveness analysis cannot be satisfied given our 
present knowledge of the educational process. 



Requirements for Cost-Effectiveness Evaluation 

Like other productive enterprises, schools can be said to have three 
general properties which taken together define a "production process." 
First, there .are educational objectives which can be defined as the output 
of the process; second, there are students, teachers, administrators, 
buildings, and other materials and personnel which provide inputs i^. a the 
educational process; and third, there exist techniques of combining the 
inputs in various combinations to produce the aforementioned educational 
objectives. 

Before we can compare the benefits and costj of different educational 
alternatives, we mub«- possess some reasonably reliable data on the rela- 
tionships among inputs, processes, and educational outcomes. That is, the 



task of cost-effectiveness analysis requires knowledge of the physical 
relations between inputs and outputs. Unfortunately, this information 
does not exist, even in. a rudimentary form. 

. The inability to carry out cost-effectiveness analyses without these 
data can best be illustrated by presenting an example of how cost-benefit 

I 

anal'^rsis has been applied to an area where it has yielded ouccefisful 
results, the evaluation of water-resource projects. __/ 

_ • 

/ Cost-benefit analysis has been used by the Corps cf Army Engineers for 
”” niore than 30 years. For a discussion of techniques used to evaluate 
water-resource projects, see John V. Krutilla and Otto Eckstein, 
Multiple Purpose River Development (Johns Hopkins Press, 3958). 

Assume that we have a limited budget which must be allocated to that 
set of projects which will maximize benefits. Before the cost-effective- 
ness analyst sharpens his pencil he is given data on each of the proposed 
projects. Thus, he would be given information on such inputs as the 
amount of dredging required, the size of and specifications of the dam 
that will be built, and so on. He is also given a set of probable outcome 
for each project j the degree of flood control, dimensions of the body of 
water that will be created by construction of the dam, hydroelectric out- 
put, and so on. 

Given this information the cost-effectiveness analyst can estimate 
the value of social benefits that derive from reduced flood damage, new 
sources of water for irrigation, industrial, and drinking purposes; pro- 
duction of hydroelectric power, recreational potentialities and other 
outcomes. He can compare these with the initial and operating costs for 
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the project and any indirect costs. Having taken account of the Benefits 
and costs of all projects, he can select that combination of projects 
which will maximize the returns to society for any given cost, What is 
important to note in this example is that the physical relationships be- 
tween inputs and outcomes are given to the analysts by hydrologists, 
civil engineers, and other water resource experts. It is these basic data 
which are required before the economist, statistician, or operations re- 
searcher can estimate the resultant costs and benefits. Why is it that 
comparable information on the schools is not readily available? 

Outputs 

The prime difficulty in evaluating outcomes of schooling derives from 
the multi-dimensional aspect of education as well as severe inadequacies in 
our abilities to measure even single dimensions of output. Anyone who 
seeks to list educational objectives is faced with a bewildering array of 

goals that are claimed for the schools.^/ While some of the goals are 
>■ 

^ ' ■ I II II I I II ^ I w 

/ See, for example, Benjamin S, Bloom (ed.), Taxono. y of Education Ob jec- 
” tives Handbook I: The Cognitive Domain (New York: David McKay Company, 
1956 . 



straightforward, others appear to be vague, and almost all of them defy 
measurement. Certainly the schools are expected to provide students with 

I • 

adequate knowledge of and literacy in language skills, mathematics, sciences 
and social studies. Then there are the more esoteric objectives of incul- 
cating a set of common values, civic pride, patriotism, appreciation of 
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culture and aesthetics, and so on. It is often stressed that the schools 
are also responsible for preparing students to properly assume adult roles 
in our society. In actuality, the words describing these objectives tend 
to lull one into thinking that the objectives can be easily defined, Un~ 
fortunately, experience has shov7n that the specific goals which are masked 
by these descriptions are neither readily evident nor are they measurable. 
Accordingly, the outputs which have usually been chosen for an anal- 
ysis of school effectiveness have been standardized achievement scores, 
dropout rates, expected lifetime earnings, and rates of college attendance 
among high school graduates ,__/ To the degree that these measures mirror 



_/ For a discussion of outputs see Jesse Burkhead £l,, Input and Out - 
Jgut iji Large-City High Schools (Syracuse; Syracuse University Press, 
1967), Chaps, I-II, For some original work idiich discusses the effect of 
compensatory education programs on the alleviation of poverty within a 
cost-benefit framework, see Thomas I, Ribich, Education and Poverty 
(Washington, D.C,: Brookings Institution, 1968), And for a cost-benefit 
analysis of dropout prevention see Burton Weisbrod, "Preventing High 
School Dropouts," in Measuring Benefits of Government Investments , Robert 
Dortman (ed,) (Washington, D,C,-: The Brookings Institution, 1965), 
pp, 117-167. 



to a great extent the ostensible successes or failures of individuals in 
middle class society, they are certainly important criteria. Nevertheless 
they may merely reflect that part of student behavior which is measurable in 
some gross sense. Unfortunately, like the visible portion of the iceberg, 
the greater part of educational output may be hidden from view. 

Yet, one salient feature of educational output must be emphasized 
strongly. No matter which type of outcome one wishes to measure, the goals 
of the schooling process must be viewed as those of changing the potential- 
ities, proficiences, and attitudes of the students who enter that process. 



That is, schools are expected to change people in socially desirable ways 
(and one might also wish that the reverse were true) . Students enter the 
schooling process as a relatively raw material, and the schools as well as 
concomitant family, community, and other influences transform these 
students into vzhat might be characterized as ”more-nearly finished*’ 
products. Thus the output of the school must be conceived of as the 
value-added to its student input -- the difference betvzeen the value of 
the student output and student input. 

In this sense the absolute levels of high school dropouts, college 
attendance, or achievement scores should not be used as measures of school 
output. Rather, the effectiveness of a school along these dimensions must 
be gauged by its success in decreasing dropout rates, increasing educa- 
tional motivation, and increasing achievement among a given set of studt.nts 
In particular, students of lower socio-economic status generally enter the 
schools less well prepared and with less educational support from their 
families than their middle-class counterparts. Even if the same amount of 
change were to take place among both groups of students, the disadvantaged 
students would show lower proficiencies at the end of the educational pro- 
cess. Therefore, it is the "value-added in performance, attitudes, and 
other behavioral dimensions that must be related to schooling inputs. Un- 
fortunately, many commentators appraise the performance of schools on the 
value of the student outputs alone, without taking account of differences 
in the proficiencies of incoming students. 

Simply taking account of the social class of students in looking at 
differences in achievement levels -- as some researchers have done -- does 
not solve this problem.__/ Indeed, outputs must be directly measu-ued as 
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_/ See the discussion on this point in S'amuel S. Bowles j "Towards an 

Educatifinal Production Function," A paper presented at the Conference 
on Research in Income and Wealth, University of VJisconsin (November 15, 
1968), pp. 23-27. 

t 

0 

1 

changes or growth along the pre- specified cognitive and affective dimen- 
sions. One way of doing this given the ordinal nature of test and atti- 
tudinal instruments is to measure the relative differences on a normalized 
scale at two points in time* That is, a school might have students who 
score at the 38th percentile on a standardized test of reading compre- 
hension at grade 4, and at the 51st percentile in reading comprehension at 
grade 6 (using national norms for similarly constituted student samples at 
each grade level). The relative position of other schools might deteriorate. 
This approach represents a crude approxmimation of relative value-added, 
though it tells us nothing about absolute gains and is subject to all kinds 
of testing aberrations .__/ 



Burkhead used a variant of this technique in the work previously cited. 
Unfortunately, problems with his data, statistical approach, and spec- 
ification of inputs prevented him from deriving useful results. See 
Input and Output in Large-City High Schools , See pp. 53-56, 71-73, and 
83-84. An example of the problems inherent in using test scores as 
measures of output is the fact that the test scores will improve simply 
by taking tests more frequently. That is, students an develop test- 
taking abilities that will improve scores. Holding other things constant, 
schools with extensive testing programs would show larger relative gains 
than those with more modest programs. 



One value-added measure that seems useful is the increase in pro- 

.ductivity and lifetime income attributable to additional years of schooling. 

Even after adjusting for differences in students’ abilities and opportuni- 

0 

ties it appears that additional schooling yields substantially higher earn- 
ings to the average individual .__/ Yet, we have no information which X 70 uld 
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/ Sec Gary S. Becker, Human Capital (New York, Columbia University Press, 
1964 ). 



relate changes in the quality of schooling and the implementation of 
particular instructional techniques to changes in productivity and life- 
time earnings. One possible approach is that of relating instructional 
techniques to changes in standardized achievement scores and subsequently 
relating differences in standardized achievement scores to differences in 
earnings (after accounting for such intervening factors as the returns to 
higher levels of schooling.) J Much work needs to be done in the area, of 

/ This technique might be used in a crude form with the recently avail- 
able follow-up and earlier set of data collected by Project Talent. 



specifying and measuring outputs. 



Production of Schooling 

Assume that we could measure educational output satisfactorily. The 
next goal would be to find out how these outputs are produced; satisfying 
this objective we could specify inputs; and finally we could assess costs 
and benefits. Unfortunately, even given a specified outcome, we do not 
know how it is produced. That is, the complexity of the world in which 
education takes place has thus far prevented us from inferring specific 
and reasonably predictable relations betX'ireen educational strategies and 

educational results. 



t 
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Indeed the basic lack of specific knowledge on how learning or 

• . 

attitude formation takes place represents the greatest obstacle to cost- 
effectiveness analysis. In the case of water resource projects we know 
that by building a dam, some fairly predictable outcomes will ensue. In 
the case of education we have the problem that even if we arbitrarily 
select and measure both outputs and inputs, there are enormous difficulties 
in ascertaining how changes in any specified set of inputs will change out- 
puts. The interactions among inputs and outcomes are so complex, and 
psychological processes are so little understood, that even sophisticated 
statistical techniques, the best available data, and the most advanced 
computers have not been able to reveal answers in this area where we lack 
basic theory.^/ 



/ For an example of the controversy on the validity of evidence derived in 
orxe of the largest studies on the determinants of scholastic achievement 
see, James S. Coleman, Equality of Educational Opportunity , Chapter III; 
Samuel S. Bowles and Henry M, Levin, "The Determinants of Scholastic 
Achievement," The Journal of Human Resources (Winter 1968), pp. 3-24. See 
also the communication by Coleman in the Journal of Human Resources 
(Spring 1968), and those by Marshall Smith; Glen Cain and Harold Watts; 
and Bowles and Levin in the Journal of Human Resources (Summer 1968) 



Thus, not knowing how schooling is produced we probably know even less 
about the specification and measurement of educational inputs than we know 
about outputs. It is true that we can measure per pupil expenditures, pupil 
tetacher ratios, physical facilities, and so on. Yet, if we have learned 
anything about the learning process it is that we must measure inputs in 
relation to their interactions with students. By this standard, it is the 



quality of the nriiiie inputs that we must be concerned with as much as 
their quantities. • * 

In the present schooling process , teachers represent about 70 percent 
of the current budget, so we must be especially concerned with the teacher 

I 

input. The standard meaFvcres of teacher input tell us little about the 
quality of interaction betv^/een student and teacher. The traditional 
measures have been class size, the teacher degree level, and teaching ex- 

I 

perience. These measures have been used because of their visibility. Yet, 
little evidence is available that shows any relationship between such 
measures and school effectiveness. 

Even fairly substantial reductions in average class size do not seem 
to have improved the quality of teacher- student interactions ._/ Moreover, 



__/ For strong evidence on this point in the light of drastic reductions in 
class size and student/teacher ratios see David J. Fox, "Expansion of the 
More Effective School Program," Evaluation of New York City Title I Edu- 
cational Projects 1966-67 (New York: Center for Urban Education, 1967), 
pp.' 32-44. 



such factors as the degree level and experience of teachers vary so much in 
quality among teachers that they, too, have rarely shown any relation to 
school effectiveness. 

Recent evidence suggests that those teacher traits which have not been 
readily visible such as attitudes and verbal facility show stronger associa- 
tions with student achievement than do any of the former measures . / 



/ See for example the evidence in Henry M. Levin, Recruiting Teachers for 
Large-City School s ('WashingtDn, The Brookings Institution, forthcoming.) See 
also, Samuel Bov/les, _op. cit . 
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Moreover, experimentation is just beginning to denote those particular 
teachers* attitudes, that seem to affect the performances of students " 



/. See for example, Robert Rosenthal and Lenore Jacobson, Pygmalion in the 
j Classroom (New York: Holt, Rinehart, and Winston, 1968). Also see 
‘ Ned A. Flanders, Teacher Influence ^ Pupil Attitudes and Achievement , 
j U. S. Department of HEW, Office of Education Cooperative Research Mono- 
• graph No. 12 (Washington, D.C.^ U. S. Government Printing Office, 1965). 



Yet, our state of knowledge is still too sparse to be able to accurately 
specify and measure those teacher attributes which are most highly related 
to educational outcomes. 

s 

Further, non- teacher inputs seem to be measured in very naive ways. 

For example, the presence of certain facilities or the age of a building is 
considered to be a reasonable measure of physical schooling facilities. 

This approach ignores the relevance of physical inputs by measuring them in 
such a way that student-facility interactions are ignored. For example, 
the presence of science laboratories tells us little about the quality or 
quantity of science instruction. Most high schools contain science labor- 
atories as a standard part of their institutional design. Yet these labor- 
atories vary in their equipment, their extent of usCj, and the degree to 
which they are integrated into the science program. In many schools science 
laboratories exist but are rarely used because teachers with science train- 
ing are in short supply due to existing salary policies. Yet a recent 

See my Recruiting Teachers for Large-City Schools . 



survey by the U. S. Office of Education used the presence of science labor 
atories as one of the two facilities measures that might affect verbal 
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achievement. / 



James S. Coleman e^. aJL, , En^^lity of Educational Opportunity , U. S. 
Department of Health, Education, and Welfare. (Washington, D.C.: 1966). 
i On the basis of science laboratories and library books alone, the 
Report concluded that school facilities show little or no relation tr 
achievement. See p. 316. 

«□ ■■ ■! I P > m ■ ■ ■. i * li ■. i.mI 

Likewise, the existence of educational television or some other form 
of instructional technology must be analyzed as an input in the way that it 
interacts with students and faculty. Simply specifying that some schools 
used closed-circuit television and others did not tell us little about the 
schooling process. That is, the existence of a physical facility tells us 
little about the extent or quality of its use. 

In short, just as outputs are highly speculative and poorly measured, 
we have very little available information oii inputs into the schooling 
process or the learning process. Attention has been devoted to measuring 
quantities of visible inputs with little attention devoted to their quali- 
ties; and emphasis has been focused on the presence of certain visible 
features of schools rather than asking the more basic question of what con- 
stitutes the schooling process. 

But the job of the cost-effectiveness analyst is that of evaluating the 
costs and benefits of different strategies. Information on inputs, 
processes, and resultant outcomes are the necessary prerequisites with 
which he must work, and the delicate job of translating this morass of 
information into a set of alternatives that can be readily compared by the 
decision maker is his science. If his information requirements are not 
satisfied, he cannot satisfactorily execute his task. 






Some Confusion 



y^t, so-called cost-effectiveness analysis is taking place in educa- 
tion with far too much claimed for it at this stage. One such effort has 
stated that it can translate expenditures on Title I programs into all kinds 
of educational outcomes including increases in lifetime earnings. __/ While 

♦ 

_/ See Abt Associates, Inc., "Design for an Elementary and Secondary Educa- 
tion Cost-Effectiveness Model" Report on the Mathematical Design Phase, 
Contract OEC 1-6-001681-1681, (U. S. Office of Education: February, 1967) 



no mechanism is shown for accomplishing this formidable task, the non- tech- 
nical reader is seduced by flow diagrams, mock computer printouts, algebra, 
calculus, and a discussion of properties of statistical functions. Un- 
fortunately, none of these solve the difficult problem for which the re- 
port claims it has the framework for solution. 

Another report on cost-effectiveness analysis estimates actual benefit 
cost ratios without knowing the true costs of programs, what the programs 
consist of and their effects on student outcomes. That is, it never ex- 
amines the ingredients of the programs; it merely assumes a causal link 
between changes in aid-to« the-disadvantaged and any improvement in their 
performance level over a period of time. _/ 

__/ See. Robert Spiegelman, j "A Benefit/Cost Model to Evaluate Edu- 

cational Programs," Stanford Research Institute (January 1968). 



Yet, I am optimistic enough to believe that many of these problems will 
be substantially overcome, and cost-effectiveness analysis will soon yield 



benefits greater than its own costs. Recent evidence, for example, has 
suggested that recruiting and retaining teachers with higher verbal facil - 
ity is five to ten times as effective per dollar of teacher expenditure in 
raising achievement scores of students as the strategy of obtaining and 
retaining teachers with more experience. 



_/ See Henry M. Levin, "Cost-Effectiveness Analysis and Educational Policy -- 
Profusion, Confusion, Promise," A paper presented to the 34th National 
meeting of the Operations Research Society of America, Philadelphia 
(November 8, 1968). 



But before meaningful progress can be made we must be honest about 
the current state of the art as applied to education. Anthony Oettinger has 
stated this point succinctly; 

"The systems analyst . . . owes it to the ideal' of professional 
integrity to tell his client the truth as he sees it, not as 
the client would like to hear it. He may or may not have a use- 
ful prescription. Polio has been conquered, a mumps vaccine has ' 
just come out, but the comn^on cold and cancer are still with us." / 



_/'The Myths of Educational Technology," Saturday Review (May 18, 1968), p. 91 

The point is that we must recognize the formidable height of the barriers 
placed in the path of cost-effectiveness analyses in education rather 
than pretending that our hurdling ability is sufficient over any course. 

It is only by recognizing the magnitude of the hurdles that we will be 
able to surmount them rather than stumbling into them headlong. 



